Skip to content

feat: automatically rotate SDK API tokens#445

Open
timmarkhuff wants to merge 34 commits into
mainfrom
thuff/gl-1709-token-auto-refresh-independent
Open

feat: automatically rotate SDK API tokens#445
timmarkhuff wants to merge 34 commits into
mainfrom
thuff/gl-1709-token-auto-refresh-independent

Conversation

@timmarkhuff

@timmarkhuff timmarkhuff commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Overview

This PR adds SDK support for automatic token rotation. The server-side support for token rotation already exists, however the frontend changes that allow a rotating token to be created aren't merged yet, so there will be no immediate effect of releasing this version; Groundlight clients will continue to function as they always have until Reef allows users to start creating rotating tokens.

Summary

  • Automatically rotate SDK API tokens when the configured token's identity has a non-null token_ttl.
  • Non-rotating configured tokens (token_ttl=null) are used as-is: no mint, no on-disk cache, no background refresh thread (same behavior as pre-rotation Groundlight).
  • For rotating tokens: mint a working child (server-authoritative lifetime; create omits client expires_at), and refresh every token_ttl / 30.
  • Working token state is stored in a per-configured-token-snippet JSON cache on disk (current + previous), updated under a file lock so multiple processes on the same machine share one rotation chain instead of each minting independently.
  • On refresh, revoke the old previous (best effort), demote current to previous, and mint a new current. Keeping the demoted token alive until the next refresh is the grace period: siblings still holding the old secret keep working for roughly one refresh interval.
  • Regenerate the API token client from the updated OpenAPI spec (by-snippet lookup, create, delete, list).

Companion PRs

This SDK release is a prerequisite for turning on finite identity Token TTLs in production via the ApiIdentity Reef work:

Those PRs are intentionally still drafts. They are substantially complete, but we will keep testing them and only merge after this SDK version is live, so customers (and Edge/hubs) that start receiving rotating tokens can refresh safely.

Please review this PR together with #6650 and #6651 so the client rotation behavior and the identity management surface stay aligned.

Related backend hygiene: axon-groundlight/zuuul#6681 nulls out legacy backfilled token_ttl values so existing never-expire tokens stay static until users opt in.

Rotation policy (client)

  • Rotate iff identity token_ttl is non-null (from by-snippet / mint responses). No environment-variable override.
  • Refresh cadence = token_ttl / 30 (e.g. 30-day Token TTL => daily refresh).
  • Per-token expires_at is still used to decide whether a cached secret is still usable, but not for cadence. Hard-deadline clamps therefore cannot accelerate refresh.
  • Identity policy is applied by the server on mint; the client only reads token_ttl and expires_at.

Dependency

  • Requires public /v1/api-tokens token_ttl on list / create / by-snippet (zuuul#6652, merged; live on integ/eksdev, not yet on prod as of 2026-07-24).

Test plan

  • Unit tests for null token_ttl (no rotate), hard-deadline-only (no rotate), cadence from token_ttl (not clamped expires_at), previous-parking, cleanup, and non-positive token_ttl clamp
  • Generated API token client coverage including token_ttl
  • Live check against api.dev with a short identity Token TTL (observed rotation with token_ttl_seconds=180, cadence ~6s)
  • CI green on latest commit (currently blocked: prod docs tests fail because prod responses omit token_ttl)

Rotate short-lived working tokens through a locked disk cache while preserving bootstrap credentials for recovery.

Co-authored-by: Cursor <[email protected]>
@timmarkhuff timmarkhuff changed the title Add automatic API token refresh feat: automatically rotate SDK API tokens Jul 9, 2026
Tim Huff and others added 28 commits July 9, 2026 11:59
Keep the new close method from being auto-registered as a shell command so CLI initialization and tests continue to work.

Co-authored-by: Cursor <[email protected]>
Back off after failed refreshes, preserve pending cleanup metadata, and make 401 recovery safe for streamed and raw HTTP requests.

Co-authored-by: Cursor <[email protected]>
Fall back to the bootstrap token when a server does not yet expose token management, preserving compatibility during deployment.

Co-authored-by: Cursor <[email protected]>
Replay generated stream bodies correctly, cover note requests, and make locking and shutdown safe for in-flight refresh work.
The bootstrap token has only one job: mint the first working token.
Keeping it alive afterward is a security risk -- a leaked bootstrap
can silently mint tokens indefinitely. Revoke it immediately after
the first slot is written.

Consequences:
- 401 recovery no longer falls back to bootstrap to re-mint; it
  adopts a fresher cached token from another process or raises loudly
  requiring human intervention (provision a new bootstrap token).
- refresh() raises on a missing slot rather than re-minting from
  bootstrap, since the bootstrap is gone.

Updates plan doc and Google Doc accordingly.

Co-authored-by: Cursor <[email protected]>
…ing rotation

The bootstrap token's name (suffix-stripped) is now written into the slot cache
as base_name once, at first mint. Subsequent rotations read it from the slot
instead of doing a paginated list call to rediscover the current token's name.

Key changes:
- TokenSlot gains a base_name field; old slots with no field fall back to a
  live lookup via the new _resolve_base_name helper.
- _initialize_token looks up the bootstrap token once; the result seeds
  base_name and is passed directly to _revoke_bootstrap, eliminating the
  second paginated scan that revocation previously required.
- _mint_replacement now receives base_name as a parameter rather than
  doing its own lookup.
- _new_token_name simplified: it no longer accepts None or strips suffixes
  (that responsibility moved to _resolve_base_name).

Co-authored-by: Cursor <[email protected]>
The GET /v1/api-tokens/by-snippet/<snippet> endpoint (zuuul#6579) is
now deployed. Replace the paginated list scan with a single direct
call, removing _find_token_by_snippet and TOKEN_PAGE_SIZE entirely.

- _initialize_token and _resolve_base_name both call _get_token_by_snippet;
  NotFoundException means "token not found" and falls back gracefully.
- Revert TOKEN_TTL_DAYS to 30 and REFRESH_INTERVAL_DAYS to 1 (temporary
  short values were only for live rotation testing).
- Update tests: list_api_tokens mocks replaced with
  get_api_token_by_snippet; _page helper and EXPECTED_PAGE_COUNT removed.

Co-authored-by: Cursor <[email protected]>
Keep the configured token alive for delayed cleanup after the first
working token is minted, and document the interim hardcoded testing
TTL/refresh cadence until identity token_ttl discovery lands.

Co-authored-by: Cursor <[email protected]>
Allow null last_used_at on API token responses so minting a never-used
token does not fail OpenAPI deserialization during client init, and
annotate next_previous for mypy.

Co-authored-by: Cursor <[email protected]>
Short testing refresh intervals raced with urllib3 mocks and inflated
call counts. Close the configure-time probe client and pause refresh on
the retry-test fixture.

Co-authored-by: Cursor <[email protected]>
Short testing refresh intervals left orphaned refresh threads that raced
urllib3 mocks in HTTP retry tests. Default tests to
GROUNDLIGHT_DISABLE_TOKEN_REFRESH=1 and close client fixtures.

Co-authored-by: Cursor <[email protected]>
Rotate whenever the current token is due: revoke previous, demote
current, mint a replacement. Drop grace-blocked minting that could
outrun short test TTLs, and clarify configured-token error messages.

Co-authored-by: Cursor <[email protected]>
Keep the helper for local testing only; it does not belong in the
branch under review.

Co-authored-by: Cursor <[email protected]>
Pass the server's unauthorized detail through to ApiTokenError so
expired or revoked tokens are reported clearly, and silence the
known broad-exception catches that pylint flags in TokenManager.

Co-authored-by: Cursor <[email protected]>
Never-expire configured tokens skip minting and refresh entirely.
Expiring tokens mint without a client TTL and refresh every
observed_ttl/30 from the working token's lifetime.

Co-authored-by: Cursor <[email protected]>
Replay multipart file bytes on 401 retry, derive refresh cadence from
server created_at/expires_at with a non-positive TTL clamp, and drop
GROUNDLIGHT_DISABLE_TOKEN_REFRESH so rotate-vs-not is only expires_at.

Also rename bootstrap_* to configured_*, require current cache fields,
treat by-snippet 404 as no-rotation, and simplify mint previous wiring.

Co-authored-by: Cursor <[email protected]>
Resolve generated/model.py timestamp conflict while keeping ApiToken and VLM models.

Co-authored-by: Cursor <[email protected]>
Drop an unnecessary multi-line import diff and describe never-expire
tokens via expires_at instead of ttl.

Co-authored-by: Cursor <[email protected]>
Annotate mixed multipart snapshot/file dict value types and make
created_at Optional so the lint job can pass.

Co-authored-by: Cursor <[email protected]>
Edge config and readiness endpoints do not validate API tokens, so
routing those calls through unauthorized recovery is not needed here.

Co-authored-by: Cursor <[email protected]>
Rely on the previous-token grace period and background refresh instead of
retrying OpenAPI and raw requests after unauthorized responses.

Co-authored-by: Cursor <[email protected]>
Tim Huff and others added 4 commits July 24, 2026 21:17
Use a long-enough fake token and stub TokenManager so these unit tests
do not hit the network during ExperimentalApi construction.

Co-authored-by: Cursor <[email protected]>

@tomfaulhaber tomfaulhaber left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a lot here, but it looks pretty reasonable.

Drive rotation from the server-reported identity Token TTL instead of
deriving cadence from per-token expires_at, so hard-deadline clamps no
longer accelerate mint/revoke churn.

Co-authored-by: Cursor <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants